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ABSTRACT 

. The development of a computer-based, university-wide 
testing system at Brigham Young University is discussed, as is its 
application to two large-enrollment math classes. The hardware 
configuration, software applications, general philosophy, budget, and 
personnel requirements of the system are also explained. Development 
of the system over time is explained as is the flow of students and 
faculty through the system. The organization of the two math courses 
Is described and the philosophy and constraints which molded the 
classes into their present management structure are explained. The 
interaction between the testing system and the students is analyzed, 
and problems of the course co-ordinators concerning the testing 

system are discussed. The results of the in teractioF between the 

system and the math classes is treated and comments are made 
regarding the advantages and disadvantages of such an interaction. 
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THE ROLE OF A COMPUTER BASED UNIVERSITY-WIDE TESTING SERVICE 
IN tiIe MANAGEMENT OF LARGE ENROLLMENT COURSES 

At Brigham Young University, the computer based university-wide testing 
service has come to serve as a major resource in the development of large en- 
rollment classes. In fact, it has become indispensible as a source of solutions 

r . • ...... 

to problems encountered in the growth and progress of such systems. This paper 
will describe the computer-based testing system on this campus. Then, using as 
examples two mathematics programs, will discuss in detail how this system can 
relate to course testing and management for very large enrollment courses. Since 
these courses have a combined enrollment of about 4,000 students^and together\^_ 
^Hte nearly 35,000 tests per semester, they have provided both the respective 
coordinators and the testing system with ample problems and situations for re- 
search and development. 

Development and General Description of SCOUT 
. Development ; - 

Testing Services (tsT) at Brigham Young University (BYU) was organized in 
the late 1950*s to provide psychological and vocational test support services to 
the Counseling Center. By 1966, the system was providing. general test scoring 
services to the entire university. In 1971, the developers of this system felt 
that 1f faculty members could have the option of testing their students outside 
of normal lecture hours, more hours could be devoted to instructional purposes 
and so began the service of providing out-of-class computer assisted testing. 
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TST management, with University administration support, thus began a 
systems analysis which culminated in the development of an entirely new method 
of test processing at BYU— SCOuf (System for Cpmputer-assi sted On-line University 
wide £esting). 

Hardware Configuration 

SCOUT is supported by a 64K, 50 megabyte Microdata minicomputer. Attached 
to the mini are ten ADDS video display terminals each equipped with a Monarch 
2246 barcode reader. An NCS 7001 optical scanner is backed up, off-line, by an 
NCS 7005 optical scanner. Completing the system are a Centronics 6600 upper/ 
lower case system line printer, a Printronix 300 1pm matrix printer, and a 
9- track, 800 bpi tape drive. Hardware costs totaled approximately $100,000. . 
SCOUT is written in an extended version of BASIC. The system took about twelve 
—man-months Hf labor" to design 7 code, de-bug, and put into operation. 

Staff, Budget, and Volume o 

The Testing Center is housed on the main floor of the University's Harold . 
B. Lee Library. It occupies approximately 6,000 square feet of floor space. It 
is staffed by three full-time administrative/ staff personnel and 25-30 part-time 
student employees. The facility is open 12 hours per day Monday-Friday and four 
hours on Saturday. 

The Center operates on an annual $200,000 budget and .is self-supporting in 
as much as academic departments and students are charged for tests administered. 

Since the implementation of SCOUT on August 26, 1976, oyer 1,200,000 tests 
have been administered. The current volume is more than 300,000 tests per year 
serving 260 different classes ranging in size from 25 to 2,500 students. 
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General Description of Services 

'There are several standard services offered by the Testing Center, vari- 
ations of which can be chosen by program coordinators or instructors for their 

classes.. / • « / 

/ 

Class Initialization . 

Class Initialization is the method by which the Testing Center is able to 
create. a class roll for each class requesting testing services. At the. begin- 
ning of a semester , before the f i rst stadent~ebmeT i Ftothe" Testing Center, the 
course instructor or coordinator submits to Testing Services a list of testing 
periods. Also provided is test weight information, telling SCOUT how a particu- 
lar test is to count in relation to all other tests given for the. course during 
-the semester;: ~The _ i7f5trucTor may request a retake option. If students aire per- 
mitted to retake an examination, 4 the instructor decides which score will count 
in assigning a grade~(either the last score or the best score). Options of . 
assigning letter grades, penalising incorrect answers, subtracting a penalty 
for late tests, etc, are available. The instructor also decides what type of 

test feedback' information is to be reported to the student. 

I 

The first step in creating a file is to access the University! s "Student 
Information System" on BYU's.IBM 4341 computer and obtain a tape of all students 
currently enrolled at the University. This tape also provides a list of -each 
student's current class registration. The tape is then used to initialize SCOUT 
class files. If a student enrolls late and 1s not included on the class roll 
when it is tapped, he can be added when he takes his first test. The class tap 
helps to eliminate ID errors and to speed up student entry at the first of the 
semester and gives course personnel an accurate class list. As soon as the tap 
is completed, SCOUT is ready for the class members to begin their testing. 
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Student Entry , . 

During the indicated testing period, the student comes into the Testing 
Center. Using the Monarch 2246, a barcode label on the back of the student's 
University ID is "read" and SCOUT responds indicating that the student is 
identified. 

/ 

i 

The Testing Center employee scans the appropriate barcode label in the test 
identification book telling SCOUT which test the student wants and the system 
responds by randomly assigning a form of the test to be administered. The appro- 
priate examination is then issued. At this time, before entry is cleared," SCOUT 
automatically checks the student's files for class enrollment. In the case of a 
retake exam, the SCOUT assesses the appropriate fee and then checks to make sure 
the assigned form of the test has not been previously taken. The test deadline 
fs^rlio^verifl^dno assure that the student is within the authorized time "period, 
and^f not, a penalty for late tests can be imposed or entry can be denied. A 
"hold file" is" checked to see if the student's file is flagged, and SCOUT refuses 
student entry until all the flags _are_cleared. - These flags-can Deactivated for 
various reasons— money owed, messages from the-ins true tor, violation of the 
University's regulations, etc. TST management can also flag a student if any 
problems exist on his cumulative test file. 

As soon as entry is cleared, the student with his exam and answer sheet is 
admitted to the testing area. This entry and clearance process normally takes 
from six to fifteen seconds, depending on the student and the test he is taking. 
Once the student is inside the testing area, SCOUT allows monitoring so that 
management has a count of all students in the system; as well as name, ID, test 
taken, and all other particulars about each student. 

Test Scoring and Student Reports 

When the student has finished his exam, he brings it to the scoring table. 
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Here a proctor puts the- answer sheet into an optical scanner. The answer sheet 
Mrpote njumte?Lj£_well as student answers are. transmitted to the Microdata for 
scoring. SCOUT* s scoring program identifies the student from the answer sheet 
ID, selects the appropriate key to score the test, and then posts the results 
in the student's cumulative file. The program also transmits to the printer 
the student's name, social security number, raw score, percent score, letter $■ 
grade, class information, etc. At the instructor's option, the program can also 
provide additional information such as: (1) a list of the items the student 
missed; (2) the student's responses to the questions; and (3) the correct answers. 
It can also compute specific area scores within the test, thus indicating expli- 
citly areas of student weakness. This process of scoring, posting and reporting, 
is completed in about twelve seconds. The student exits with a hard copy of his 
exam results. 

If the student lo$es"~h\s report, or if he wants a cumulative report, SCOUT 
can reprint his original^report or produce a report listing all tests taken for 
all classes on the system in which he is enrolled. 

On-line Cumulative Faculty Reports 

As soon as SCOUT* s scoring program has posted the score to the student's 
cumulative file, this file can be reported to the instructor. Generally, but 
not necessarily, these faculty cumulative reports run at night after the office 
is closed, and are thus ready the next morning. These reports can be alphabetic 
by class section or alphabetic overall (all sections combined). They can also' 
be printed in social security number order without names foresting purposes. 

The format for the cumulative faculty report lists percent, total points, 
and can assign a cumulative letter grade in addition to listing student scores 
on each exam. For each exam, mean scores, standard deviations, medians, modes, 
and scoring ranges are printed. Depend ingvujxm the option chosen when the class 
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was initialized, the reported score on a particular exam can be either the 

student's best or his last score. The instructor can even change this option 

and get reports both ways if desired, thus allowing measurement of student pro- 
gress on. retake exams. 

In addition to the grading features just listed, an item analysis is pro- 
vided on each test item giving frequency of response, percentage of students 
answering each response, and a co-efficient of selective efficiency to determine 
item discrimination. Score distribution means, modes, medians, standard devi- ' 
ations, kurtosis, skew, and other statistical measures are also provided to 
facilitate a more complete analysis of each test. 

Grades Transmission 

> 

Traditionally, at BYU, instructors have had the responsibility of assigning 

a student's semester grade, coding a grade roll, and submitting that grade roll_ 

-to the -Records- Office. "Thaf'bff ice Tin turn bringsj^he rolls to. Testing Services 
to be optically scanned and sent to Computer Services for processing and posting 
to the student's transcript. Inasmuch as Testing Services already had the SCOUT 
test data in machine readable form at the conclusion of the semester, a new 
module was added to the system at the Math Department's request, to allow in- 
structors the option of transmitting their grades directly. This bypasses the 
somewhat cumbersome and error-prone grade roll procedure and saves the instructor 
many hours. of tedious semester-end work. 

Testing Services and the Growth of Two 
Large-Enrollment Math Courses 

The two courses to be displayed make up roughly 33% of Testing Center 
business^ These"coarses begarr about frvVyears ago to experience phenomenal 
growth. As the numbers of students began to grow, the systems began to be 
strained. These two courses presently serve about 4,500 students per semester. 
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Most of the students are pursuing majors outside mathematics. There is a rather 

limited-tenured-faculty involvement at this level and a small graduate program 

in math does not provide the number of graduate students needed to serve the 
needs /of such large enrollment courses. Special management designs were needed 
because of the large enrollments and instructor limitation. This led to the 
/^creation of some flexible, but very stable, systems in which instructors can 
' . enter and exit' without^destroying the continuity of student progress from one 
semester to another. The instructor's concern is with content and presentation 
rather than course management, testing, etc. * * *• * 

The courses served by these systems are (1) Basic Math Review, which is a 

remedial course preparatory to the Basic Math Skills Evaluation required as a 

- '■ \ \ • 

part of the University's General Education program; and (2) College Algebra/ 

Trig, which is a calculus preparatory coufip, but J s also terminal for many. 

For both of these programs, a mastery learning format was chosen. Multiple 
form testing is used with feedback available after each test attempt. The 
students are strictly paced as they proceed through the course material. Grades 

in both courses are assigned against fixed scales. 

\ 

Math 1 OOP - Organization of the Course 

Math 100D covers the basic mathematical skills and applications needed to 
pass the General Education Evaluation (known hereafter as the G.E. exam). The 
course will provide students with: 

1. A review of applied arithmetic skills involving percentages and 
statistical graphs. 

2. A review of-common geometric figures and measurement formulas and 
an introduction to the metric system. 

3 3. An introduction to basic principles and skills in working with 
algebraic expressions and equations. ^ 

4— Expert ence~in~applying arithmetic and algebraic" skills to inter- 

* preting and solving verbal problems. 
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There is an especially heavy emphasis on verbal problem solving skills. 
This emphas'is is consistent with the goals of the general education experience. 

The course material is organized into eight (8) study units, each of which 
is followed"' by an examination. In addition, the G.E. examination serves as..a 
comprehensive final. 

For most students in this course, there are no lectures given and there 
will be no formal class period assigned. Instead, the high support component 
of this course offers students the support and resources to meet the needs of 
their' individual learning process. 

Math 110 - Organization of the Course 

Math 110 is the precalculus mathematics experience. It contains material 
often described as college algebra and trigonometry plus some additional topics. 
It is organized into eight (&) individually 5 taught and graded mini-courses on 
modules: 



10CF A Review of Fundamentals 1.0 

110A Polynomial and Rational Functions 1.0 

110B Exponential and Logarithmic Functions 0.5 

HOC Matrices. and Systems of Linear Equations 0.5 

HOD Combinatorics 0.5 

11 0E Linear Programming 0.5 

lilA Trigonometry 2.0 

11 IB Coordinate Geometry 0.5 



A specific sequence of modules arranged for presentation during the 
semester at„a fixed. hour in a given room is called a track . A track schedule , 
a description of the current semester's offering of modules listed in tracks, 
is available to each student and in the math lab. 

". A student registers for Math 110 for a variable credit of at least two (2) 
hours~(unless permission is given by the program director). After attending a 
first day orientation, he determines which modules in which tracks he wishes to 
attend. The student indicates this initial program through an initialization 
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procedure during the first days of the semester. If the student later wishes . 
to change this initial program, he may do so up to the date posted for finali- 
zation of all Math 110 programs. Note that at no time may a. student drop Math * 
110 without going through the official university drop procedure nor may a'stu- * 
dent change his program to less than two (2) credit hours in Math 110. 

A student's grade in a particular module is determined by the*- score attained 
on an examination covering the material in' that module. The student has the oppor 

* o 

tunity to be re-examined. twice over the module at his expense but all examinations 
over a particular module in a particular track must be completed by-theUate 
posted in the current track schedule. 

Specific Course Problems and Concerns 

Math 100D v 

Enrollment 

i 

Early in the development of the Math 100D program, students enrolled in the . 
course through regular registrations but enrolled at the Testing Center by taking 
their first exam. This caused some differences, to occur between" the University 
class roll and the roll reported from the Testing Center. In addition* the 
Testing roll also continued to carry the names of students who had dropped the 
class after taking one or more exams. By the end of the semester, these differ- 
ences in the rolls were the source of confusion and concern for the instructors. 
Implementation of the class tap procedures in 1978 provided instructors not only 
with current student progress, but also with current course registration infor- 
mation that was consistend ? with University rolls. 

Testing 

All testing for Math 100D takes place in the Testing Center. This allows 
. the student to choose a testing time when he is prepared and when it is convenient 
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for him to spend the necessary time. The program provided by SCOUT paces. the 
students through nine exams according to a schedule provided by the' course 
coordinator. The SCOUT system keeps track of who has tested, how many times t 
each student has tried each<exam, and records the results of eadrstudent's 
best try for instructor inspection* £ince about 17,000 tests are processed 

each semester for Hath 100D, this tracking and recording service saves hours 

* 

of instructor time which can then be used for tutorial, counseling, or develop- 

ment purposes. which are critical, in .large enrollment courses* 

In addition to the giant bookkeeping assistance, however, the Testing 

Center director and his staff have played a major roll in helping the course. 

developers and coordinators to streamline the testing process so that student." • 

can test as quickly and smoothly as possible. Their suggestions and assistance 

in problem solving have been major keys to the success of system changes. 

f 

Grading ' • • 

Early in the history of Math 100D, the computer assigned all grades except 
those altered by instructor decision. Recording of final grades for submission 
to Records was done by hand marking grade rolls. This process* was laborious and 

exceptionally prone'to human errors* Incorporation. of the grades transmission 

* u * t 

feature of 5CCUT relieve^ both the labor and the err^r§ oCJtbe grading- process. 

( * ■ • * «i k y 

tfrth Math 100D population at about 1,500 students per semester, thVrel itively 

error-free submission of grades saves an enormous apiount of beginning-of -semester 
grade change hassle. • fit * 

•• * , 

Evaluation * 

y. . • , • 

One of the strongest developmental supports offered by the SCOUT system and- 

the Testing Center is its ability to provide data and data interpretation for 

program evaluation purposes. Math 100D is populated with .students w|th a* wide 
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range of abilities and special needs. In order to serve them well, areas of 



weakness in the system must be identified and strengthened. SCOUT has provided _ 
Information about the relative difficulty of our exams. It has helped to identify 
bad questions or questioning techniques. The system has assisted in surveying 
student attitudes toward a particular part of the management system or study 
materials. Student Information such as the relationship between ACT scores and 
student success has helped in our counseling efforts. Together, the Center and * 
course coordinator have identified and developed tentative ^solution possibilities 
with great regularity. 



Because of the special nature of Math 110 enrollment, computer assistance 
is extremely helpful. In Math 110, 2,400 students enroll through usual registra- 
tion. These students are then distributed into 10,000 modules which translates. ; 

to a 10,000 student load by the process of;inftTaTization. The Testing Center 

_ " V 

scans and records all the initialization forms. Later in the term when each 
student has stabilized his program, he can retain his initial program or he may 
change his program by finalising. Again, the Testing Center scans and records 
this new information into a final program roll which is°then submitted 'to reg- 
istration for printing of grade rolls. 



Students in Math 110 accumulate about 15,000 tests during- a semester. 
This includes 10,000 first takes of each module final 'and about 5,000 retakes. 
SCOUT keeps track of student testing, paces the student according to testing 
intervals and allows him within-these intervals to try each exam three times. 
A special feature of the Math 110 program is that it actually paces the student 




Math 110 



Enrollment 
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with two sets of deadlines— an initial or "bonus" period and a final deadline. 
If the student takes and passes the test on or before the initial deadline, he 
receives a half grade step bonus (B to B+, etc.) for that test. This bonus 
then follows the student on any retake of that exam he chooses to write. The 
student may also change his testing date deadlines by changing tracks. SCOUT 
contains student track information from initialization so the track can be 
changed by using a special form from which information is typed into the system 
by a terminal operator. _ — — ~ 

Grading 

The computer chooses the highest letter grade from each module for which 
the student is registered and records the grade for transmission to records. 
The utilization of the grades transmission package in SCOUT is even more valuable 
in this course than for Math 100D. As was previously mentioned, even though the 
popuK ion of Math 110 is about 2,400 students^ because- each student is-graded 
for an average of four modules, close to 10,000 grades are submitted. Before 
the direct grades transmission was used, the human error factor here was phenomenal. 
That problem has been reduced to a small, easy to handle a number of grade problems. 

Results 

The Department of Mathematics, in spite of the expense, are in support of 
the importance of continued involvement and research in the computer assisted 
areas of the two math systems described. Early in the mutual involvement be- 
tween Math 100D - Math 110 and Testing Services, survival was the main impetus. 
The use of SCOUT centered around bookkeeping and flexible testing hours. Now 
that instructors have more time and energy available to look critically at their 
programs, the much more powerful nature of the system such as for evaluation 
efforts has become a welcome part of the experience. 
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